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Documents are available from the Educational Policy Research 
Center at Syracuse in three formats, besides the regular pub- 
lication, Notes on the Future of Education: 



RESEARCH REPORTS 

Reports which have completed review by the EPRC and which 
deal with specific, policy oriented research. The reports in 
this series are usually marked by intensive research, either 
quantified or historical, and address themselves to specific 
research questions. 



EXPLORATORY REPORTS 

Reports which, while dealing with policy issues, often approach 
the realm of conjecture; they address themselves to social 
issues and the future, may be prescriptive rather than descrip- 
tive in tone, and are, by nature, more controversial in their 
conclusions. The review of these reports by the EPRC is as 
rigorous as that for Research Reports, though the conclusions 
remain those of the researcher rather than necessarily repre- 
senting consensus agreement among the entire Center staff. 



WORKING DRAFTS 

Working Drafts are papers in progress, and are occasionally made 
available in limited supply to portions of the public to allow 
critical feedback and review. They have gone through little or 
no organized review at the Center, and their substance could 
reflect either of the above two categories of reports. 



The research for this paper was conducted pursuant to Contract No. 
OEC-1-7-070996-4253 with the Office of Education, U.S. Department of 
Health, Education, and Welfare. Contractors undertaking such projects 
under Government sponsorship are encouraged to express freely their 
professional judgment in the conduct of the project. Points of view or 
opinions stated do not, therefore, necessarily represent official Office 
of Education position or policy. 
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EDUCATIONAL POLICY FORMULATION: PLANNING 

WITH THE 

FOCUS DELPHI AND THE CROSS -PURPOSE MATRIX 



INTRODUCTION 



Our concern is to clarify and to place in perspective two existing 
devices for social policy formulation. There is a need for planning tools 
that allow multiple publics to contribute to the policy formulation process. 
Most tools for policy planning and forecasting do not address themselves 
to this charge. Two, in particular, cause us great concern: the Delphi 

and the Cross-Impact Matrix. This report details their purported func o 
demonstrates the fallacies of their claims and details modifications we 
have designed and tested to turn these devices into tools for policy-makers 
and institutional planners. We call these modifications the FOCUS DELPHI 
and the CROSS-PURPOSE MATRIX . 

The Focus Delphi is a tool to help planners systematically identify 
existing consensus and dissensus among various groups in society. The 
process systematically collects the perceptions of an array of interested 
publics and keeps separate by defined group their responses about the goals 
and needs under examination. It delivers, among other things, estimates 
about value to self and/or society, consequences foreseen, perceptions of 
power to cause the event, and strategies possible to enhance or retard the 
occurrence of the goal . 

Knowledge of the. opinions held by various publics is important when 
trying to define where we are and where we wish to go. The different 



perceptions of where we should go (our several goals) may help responsible 
men to decide priorities more equitably when making policy for the universe 
of their influence. It is necessary to understand in some detail the pro- 
posed strategies of the several goal advocates in assessing institutional 
priorities. The Cross-Purpose Matrix is designed to bring these strategies 
into the open in order to aid priority assessment. 

This process arrays the competing goals in a matrix and forces, through 
human deliberation and discussion, an examination of the relative value of 
each with reference to all others. The resultant display is useful when 
priorities must be set based on limited dispersible resources. 

The devices are dealt with together in this paper because we feel they 
are related. Both were designed at the Educational Policy Research Center 
at Syracuse by the author and have been used by several groups over the 
past two years for similar purposes. 

I wish to make two points very clear at the beginning: This is a 

report of work that spans the last two years. Much of what is reported 
out here has appeared in bits and pieces in reports of on-going work with 
these tools. There is no claim made or implied that these devices as out- 
lined will have value to the reader if translated directly. This is a 
st ate-of-th e-work report. I am making this report available to speed its 
examination, utilization and modification. I request that all work that 
emerges from a reading of this report be brought to my attention so that I 
may be kept informed of its development and in turn share with interested 
readers my developing work. 

A second point is in order. As with all devices that aid the decision- 
maker in collecting information, these two — more than most others — have great 
potential for misuse and manipulation. I can only hope that they will be 
used with integrity. 
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PART I 



THE FOCUS DELPHI: A TOOL TO QUERY INTEREST GROUPS 





THE FOCUS DELPHI: A TOOL TO QUERY INTEREST GROUPS 



The Delphi Technique 



Claims 



The Delphi Technique is a process for eliciting and refining the 
opinions of a group of individuals. The individuals remain anonymous to 
each other, their opinions are continually refined and reiterated, and 
feedback to participants is controlled. The process is believed to pro- 
duce either converging group consensus or polarization of views (Gordon, 
September 1969, p. 3; Helmer, September 1969, p. 2). 

The Delphi Technique is a variant of the panel or committee approach 
for arriving at a consensus or majority opinion. Its design eliminates 
face to face confrontation as experienced on a panel or committee. Pre- 
sumably, it prevents specious persuasion, individual unwillingness to 
abandon publicly held positions, and the bandwagon effect of a majority 
argument. It replaces direct discussion with a series of carefully con- 
trolled questionnaires that return edited opinions and new information to 
the participants, who then act in privacy and respond to the successive 
inputs. The committee or panel report is replaced by tabulated data from 
the several respondents, from which the Delphi coordinators make their 
interpretations and analyses to arrive at a series of forecasts, opinions, 
and occasionally scenarios, rather than an expository report. 
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The major proponents of the technique— Norman Dalkey, Theodore Gordon, 
Olaf Helmer — accept the assumption of the decision-maker as purchaser of 
expert opinion. They argue that when questions asked are in the long- 
term future, are complex in nature, and are subtle in their relationships, 
one expert is inadequate. They assume that many experts are better than 
one and that experts as a group will exercise careful judgment in their 
deliberations. A consensus is sought, rather than an array of self- 
interested responses. 

They also speak of the method as a proven technique (Gordon, Septem- 
ber 1969, p. 3) which generates consensus when long-term forecasts are 
iiivol' ~'d. Further, it is claimed that the output of a Delphi provides the 
basis for subsequent planning, action, and analysis (Gordon, September 
1969, p. 4). 

The purposes of the Delphi as it exists are believed to be: 

- the collection of expert opinion; 

- the elimination of certain negative aspects of panel 
responses ; 

- the generation of consensus opinion about certain events; 

- the generation of greater or more reliable information 
than any one expert could produce. 

Delphi can ostensibly be used as: 

- a tool for forecasting; 

- a basis for long-term planning, action, and analysis; 

- a device for eliciting careful judgment, 

- a seemingly coherent structure for testing alternative 
contemplated actions. 

The device has been used as a forecasting process leading directly 
to plans in the areas of political alliances, technological potential, 
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war prevention techniques, economic indices, medical developments, and 
education. (Gordon, September 1969, p. 4) 

Process 



The basic Delphi process involves a series of questionnaires. The 
first questionnaire is a list of future events chosen by the coordinators 
of the Delphi and/or their employers. This questionnaire is distributed 

I ! 

to the participant experts chosen by the coordinators and/or their employ- 
ers. Each participant is asked to indicate both when they think the event 
will take place and how familiar they are with the item, then to suggest 
additions to the list of events. The estimate of the time horizon for 
the event involves a probability estimate as well. That is, for each 
event the respondents supply a date for 10% probability, 50% probability, 
and 90% probability, thus estimating the earliest possible date, the date 
most likely, and the date before which the event almost assuredly will 
have occurred. The Delphi coordinators tabulate the results of the data 
from this first round of questionnaires and edit the additional events 
suggested. 

The second questionnaire is similar to the first, but displayed with 
each event is the individual’s own response, and the inter-quartile range* 
of responses generated bj T the entire group of participants. Each partici- 
pant is asked to justify why his opinion is outside the inter-quartile 
range (if it is), and/or to shift his estimate if he so chooses. (In 
certain variations, the second round asks for important societal and tech- 
nological changes or consequences that may occur because of the event 
[Gordon, September 1969, p. 17] without asking the respondent to place 



Inter-quartile range: That middle 50% of all responses, dropping out 

the 25% extremes at both ends of the scale. 



the event in time.) The coordinators then tabulate a new inter-quartile 
range and edit the reasons given for opinion shifts. 

The reasons for any shifts and the new inter-quartile range are 
returned to the participants as round three. The participants are asked 
to examine the reasons given for opinions outside the inter-quartile range 
and to shift their date if they so choose. (In variants as described 
above, the process of stating opinions and reasons occurs in this third 
round along with a value assessment concerning the effect of the event 
on society.) The coordinators tabulate a new inter-quartile range and it 
is returned to the participant with his previous estimate. 

The fourth and final round asks the participant to address, for each 
event, any/all of the following questions: 

- What are some of the consequences of the event's 
occurrence? 

- What methods of control might be feasible to prolong, 
prevent, or enhance the event's occurrence? 

- What are the social consequences ('very favorable' to 
'very detrimental') which may result if the event occurs? 

The coordinators analyze this data to construct scenarios of future blocks 
of time spanning 10-25 years. 

Note: This is only a description of the most common Delphis. There is 

nothing that limits the number of passes or the questions asked. 

The terminal display of the Delphi exercise is generally a series of 
scenarios reflecting the Delphi coordinators' interpretations of the inter- 
quartile range data, a listing of the events with the final inter-quartile 
range for each, and other edited information of the participants. 
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Keeping in mind both the purposes of the Delphi and the assumptions 
that underlie it, we must ask if it is a good forecasting tool, of any 
value for planning, or merely a useful set of systematic questions. 

The Delphi Assumptions: An Examination 

Assumption: The inter-quartile range is an accurate reflector of attitude 

In the final tabulations of J he opinions generated by the Delphi, 
the only opinions dealt with r + se within the inter-quartile range of 

responses. The inter-quartile arg reflects only the opinion of 50% of 
a group. It is not a majority opinion. A panel of experts confronted 
with 50% agreement might deliver ^ majority opinion base! on the chair- 
man's vote and a minority opinion or opinions reflecting the range and 
reasons for dissent from the majority. Although not specifically excluded 
in the process, the Delphi coordinators historically do not supply minor- 
ity reports, even though the consensus reported does not represent agree- 
ment as to a date, but rather the span of time within which the middle 
50% of the respondents fell; nor can the importance of the lack of a 
minority statement be dismissed in assessing the Delphi as a forecasting 
and planning tool. 

Assumption: Expertise can be identified 

Who is an expert on the future and is a group of experts necessarily 
better than any one expert? Delphi proponents admit that general exper- 
tise about social affairs is next to impossible to locate in any one man, 
and that expert agreement ir general or specific areas is difficult to 
reach. The input to the Delphi is always drawn from a group of what are 
constituted as "experts." Expertise is a descriptor attributed to some- 
one, not a quality one can ascribe to himself. A person can claim to be 
knowledgeable about a field, while others may ascribe the status "expert" 




to him, based on their perceptions of his knowledge base. The future is 
a mental construct each individual develops in his own mind (Wolf son, 

1970 ). 

A claim of expert se about events in the long-term future may be predi 
cated on the knowledge of some goal or pi*. series of events. Without 

a knowledge of the strategies and goals of v a pi rmin^ group which asks 
the Delphi question, it would seem difficult ~cr -he x spondents to supply 
accurate judgments on, or useful focus about. is p iited in limbo. 

Assumption: Aggregate opinions reflect valid £o' casts 

Is the Delphi a forecasting tool? A foreca is an opinion about 
the future. One-half of the opinions given in = ielphi are considered 
the aggregate forecast of the group concerning an event v s occurrence. 

Assumption: Event forecasting is directly relevant to strategy implemen- 

tation 

Is it a planning tool? A plan demands the specification of a goal 
or the end to be reached by that plan. With Delphi, the plan is not sup- 
plied to the experts before their deliberations and the events are not 
internally linked to any goal or strategy. It would seem difficult to 
assess a plan based on the opinions of men who were not aware of the plan 
prior to their deliberations. 

Assumption: Queried "experts" forecast rationally 

Can the experts be assumed to make rational judgments? No evidence 
exists for this claim. Nor does consensus of one-half of a group repre- 
sent careful judgment in arriving at that consensus. 
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Assumption: Delphic structure explicates coherent patterns 

Is the Delphi a seemingly coherent structure for testir alternative 
contemplated actions? The events are not focused against a plan and they 
are not linked to the present in any plausible way to allow planner to 
assess a strategy in light of them. 

Assumption: It is possible to respond with reasoned judgment to the 

events and the queries about them. 

It seems unreasonable to ask respondents to make a value assessment 
of an event without knowing the entire social fabric surrounding the 
moment of the event’s occurrence, their own place with respect to that 
future moment, and their personal goals for that moment. 

Focus for Modification 



In spite of all these weaknesses, we might reasonably ask if the 
questions asked and the responses generated through a Delphi do supply 
information that can be useful in long-term planning. The process 
involved in the Delphi Technique could become a valuable tool for planning 
if we assign to it different values, and reconsider the population com- 
prising the "expert” input. We know that the process focuses the respond- 
ents’ attention on an array of possible events in the future. Depending 
on the variation, it demands that they detail a date, a probability esti- 
mate, a desirability quotient, a list of perceived consequences, and an 
explication of a possible way to control the event’s occurrence. These 
opinions or perceptions about isolated events in the future may have 
value for the planner if the assessments are made with some reference to 
a plan, a strategy, or a goal. 




Most human events (i.e. , social changes, technological advances, etc.) 
are the result of human actions and reactions. Men believe that the en- 
vironment is controlled by specific actions. If we think of future events 
in terms of who might act to make them occur we might think of them as 
goals. We might sa> that future events are similar to goals in that they 
are the 'not yet occurring states of affairs 7 that to occur must be aggres- 
sed for by someone or some group. This idea may have value in clarifying 
our use of the Delphi to assess our plans. 

If we assume that goals are a type of future event , and that for a 
goal to be realized some strategy must be pursued, then we can argue that 
a specific goal can be made to occur either by pursuing a specific strategy 
or by allowing some condition to exist such that, at some point in the 
future, it will be likely that the event will occur. Obviously, simply 
allowing a condition to exist is a strategy if we consciously recognize 
the outcome it might produce. 

To know that an ervent, X, may occur in the future and that the event 
may occur between now and the moment of occurrence of another event, Y, 
is not sufficient information to judge the impact of event X on Y. We 
must know the strategies to Y that will be operational at the moment of 
X to estimate an impact. We might say that the strategy followed in the 
pursuit of Y is comprised of a series of other, secondary, events /goals . 

It is these interim events that can give us the perceptual base to make 
opinionated judgments about specific events that we examine with a Delphi 
process . 

From the above, four assumptions can be stated which form the basis 
of a useful redesign of the Delphi process. 

1) Plans, goals and strategies for a society affect different sub- 
sets of that society in different ways. 





2) Those self-interested subsets might be reasonably perceived 
• no the several ’’experts" about the plan, strategy, or goal. 

3) It is against the proposed actions of a society that societ. 
events must be weighed. 

4) Men in the aggregate don't make plans and pursue strategies 
toward goals; rather, individual men representing individual 
institutions do. 

Without seriously altering the process of the technique we might 
alter the nature of the inputs and the populations from whom we leques 
inputs. The persons in a planning department are certainly expert con- 
cerning their personal goals for the institution. Their opinions must 
be integrated into the feedback system the Delphi supplies. The expet ;s — 
that is, those persons comprising the respondent population — might be he 
several populations who are in some way involved in, or affected by, t ,e 
pursuit of some goal. All persons are experts in their own opinions 
about the value of some goal. All persons are experts in their own 
opinions about the value of any goal to themselves. Different persons 
or interest groups may have radically different opinions. Thus they 
cannot be seen as an aggregate from which one might extrapolate directly 
a consensus opinion about events relevant to those goals. To disaggre- 
gate the several subgroups, the inter-quartile ranges of the several 
expert/interest groups could be displayed to allow the planners to assess 
Lne differing perceptions of those subgroups. 

Further, goals are' pursued through strategies. For any event, some 
group or groups have the power or authority to instigate a strategy which 
will greatly increase the probability of the goal's occurrence. The 
opinion of individuals involved in a Delphi exercise may be predicated on 
their perception of the probable actions that must be taken by men with 
authority to cause the event's occurrence. 




If the last question asked of the several expert /individuals (What 
controls might be possible for this event?) was replaced with: "What 

subgroup of those involved could control the occurrence of the event and 
how?" we might generate information for planners that would help them in 
their resource allocation. In pursuing a goal, certain affected sub- 
groups may have to be convinced of the merits of the goal. They may hold 
a misperception about the goal or the strategy to the goal, and will thus 
affect the goal's attainment. Energies may have to be Invested to con- 
vince these groups.* A knowledge of the perceptions of the several groups 
and the opinions about the power base for each event in question might 
focus the planner's attention and energies in those areas where the most 
opposition or misunderstanding exists. It may also conserve energies by 
identifying where they are not needed . 

These modifications point to an altered use of the Delphi only hinted 
at by the designers. That is, Delphi can generate a framework of goals 
from which institutions and the publics they serve can choose priorities.** 



The Focus Delphi 



The Focus Delphi is a modification of the Delphi technique. Its 
specific purpose is to translate a forecasting tool into a device to 
assist planners. 



The first assumption underlying the Focus Delphi is that planning 
addresses itself to the pursuit of goals. Further, goals cannot be asses- 
sed in isolation; rather, they must include a description of the strategies 



Or, alternatively, to get them to explicate their, possibly more valid, 
reasons for disagreement. 

For a more thorough and comprehensive analysis of the Delphi Process, 
see Weaver, W. T. , EPRC 1969-1971. 
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to be pursued to arrive at the goal. The original Delphi process holds 
society as a constant so that quantitative information can be generated 
for forecasting. While it may be sufficient to know when an event might 
occur for forecasting, the social planner cannot afford to treat people 
as a constant. He must address their concerns. To do this, he must tap 
their perceptions of the relative value of the goal and note the. strate- 
gies they might bring to bear to enhance or retard the occurrence of the 
goal, before commencing action toward either his plan or his goal. 

A second assumption underlying the Focus Delphi is that goals are 
a class of future events. While the Delphi treats future events as inde- 
pendent entities, an advocate can be identified for goals. This advocate 
is someone who has the responsibility or the desire to see the goal occur. 
The motives of the goal advocate are interwoven with the goal itself — 
as are his strategies. A planner needs to know how people from various 
segments of the affected society might react to the projected goal and 
its strategy in order to define his plan. 

A third assumption of the Focus Delphi is that for any institutional 
goal, not only the advocates, but also the affected populations can be 
identified. Each group would certainly have opinions and beliefs about 
any goal, each for differing and legitimate reasons. 

The Focus Delphi deals with these assumptions. It is conducted in 
the same way as the Delphi, with several anonymous rounds of reiterated 
data, searching out but not forcing consensus information. The differences 
from Delphi are substantive; they are differences in questions asked and 
data dealt with, rather than differences of process. 

First, participants are drawn from each population and interest 
group affected by the goal, not from some "expert" group. Second, each 
group has its own inter-quartile range displayed on reite tion, rather 
than a blending of estimates for all groups. Third, the questions in the 



third round deal with perceptions of power (i.e., Which of the involved 
groups has the power to cause the event? and. What strategies might you, 
as a member of your affected group, bring to bear to enhance or retard 
the event's occurrence?). 

With these modifications, the planner can now elicit, not consensus 
but sustained divergence, to help him assess his plan. Second, the process 
is appropriate to meet a yet unsatisfied need — a way to gather resource 
information for reasoned and informed policy-making. 

An Experiment With the Focus Delphi 

The first Focus Delphi was conducted for the New York State Education 
Department Bureau of Two-year College Programs in 1969-70 under the direc- 
tion of DeLayne R. Hudspeth. What follows describes the structure and con- 
tent of a Focus Delphi designed to collect and report how persons both in 
and associated with an education system view events relevant to their 



The major function of the research was to refine the Delphi process 
so that those engaged in policy planning could more clearly explore social 
alternatives by analyzing levels of consensus within a social system. 
Unlike a traditional Delphi, the purpose was not to elicit, or force, 
consensus, but rather to discover where — or where not — it already existed. 
Subsequently, we looked at the input/ throughput /output characteristics of 
an educational system to determine if time estimates and value assess- 
ments differed significantly between these sectors. We also included a 
group whose relationship to the primary group was largely supportive; 
either through a consulting, funding, or policy-making role. 

Although only time can tell if the differences between groups are 
significant (statistics for futures analysis being s "ill in the future) , 
the results of the study were gratifying to the res irchers . Data for 



future 




some of the events showed a clear difference of opinion by role. In tra- 
ditional Delphi studies, these differences are combined into one inter- 
quartile range, thus burying potential conflict among various publics. 
With the Focus Delphi it becomes possible to identify those events where 
time estimates or value assessments differ between groups within the 
system. 

Knowledge of the differences of opinion held by those with different 
roles within a system is valuable for the policy-maker; identifying dis- 
agreements might lead to one strategy for an event having high probabil- 
ity of occurrence (all sectors having agreed as to when an event might 
occur and to its potential value) and another strategy for an event where 
one or more sectors clearly disagree. 

Delphi is suited for displaying a wide range of events; frequently 
the range of events is so broad that the major problem in using the data 
stems from the discontinuity among events. The problem in designing the 
Focus Delphi was to structure the data collecting system in such a way 
that events were internally coherent. It was particularly important that 
participant views could be analyzed by input category so that differences 
determined by the function of the group within the parameters of the "af- 
fected population" would be clearly visible. 

For the purpose of educational planning it is important that both a 
"producer" and a "consumer" be considered expert. That is, policies 
should be planned using at least three kinds of data: 1) about the pro- 

grams the student comes from as he enters the educational system, 

2) about the program itself, and 3) about the "consumer," the student 
and employer or next school system. In short, the input, throughput, and 
output of an instructional system must be a part of the planning process. 
Further, any planning effort must take into account present programs, and 



the trends they represent, as well as totally new programs (for example, 
environment monitoring technicians). 

Keeping the above views in mind we named this process "Focus Delphi." 
Focus Delphi: Design Factors 

Traditional Delphi studies select their participants on the basis of 
subject "expertise." These judgments are made on the basis of prominence 
in their field, reputation, professional visibility, and, to some extent, 
their being known to — or more importantly by — the Delphi coordinators. 

As detailed above, however, we felt that "expertise" was of less value 
for planners than a range of affected interest-group attitudes. The 
participants, therefore, were not assumed to be familiar with Delphi nor 
to possess the conceptual attitudes required in completing a traditional 
Delphi questionnaire. For this reason, the questionnaires had to be as 
simple as possible, and the information feedback as limited as was conso- 
nant with practicality. 

Four groups were defined: high school personnel who sent students 

to two-year programs assuming something went on there; college faculty 
and administrators who thought they did something; personnel officers 
for corporations who hired graduates of these programs thinking that 
they were buying something; and finally, a loosely defined group of 
policy advisors, legislators, and consultants, who comprise an influence 
structure for education policy. 

The nature of this experimental Delphi necessitated that the events 
to be examined were generated by persons other than the Delphi coordina- 
tors. Thus, the first round of the Delphi consisted of a questionnaire 
designed to elicit from each respondent ten (10) events that he considered 



as plausible occurrences in the future. The cover letter to the question- 
naire defined what was meant by an event, explicated possible categories 
of such events, and requested participants to list them on a response 
form (see Appendix A, pp. 47-50). 

Because it was assumed that respondents would describe similar events 
with different terminology, and that these descriptions could alter time 
perceptions as to when the event might occur, the coordinators did not 
ask participants to date the events in any way. In fact, respondents 
were requested to conjecture without reference to time for this phase of 
the study. 

When all first round responses were returned, the Delphi coordina- 
tors divided them into the four respondent groups (not defined for respon- 
dents until the cover letter for Round III; see Appendix C, pp . 57-62). The 
events were then collated for similarity, edited, and rewritten. The sheer bulk 
of the number of suggested events required selection by the coordinators. 

Events for Rounds II and III were chosen on the basis of four criteria: 

a. Repeated occurrence across all four groups; 

b. Heavy emphasis within one group; 

c. Imaginativeness and/or "interestingness" (purely subjective) ; 

d. High visible impact on electro-mechanical technology and 
education c 

When the hundreds of various suggested events had been narrowed to 
a workable number (sixty in this study) , they were further organized into 
three categories : 

a. Technological innovations ; 

b. Events impacting primarily on educational institutions; 

c. Events impacting primarily on industrial institutions. 



These events were then typed on the Round II form and mailed to 
respondents (see Appendix B, pp. 51-55). 

Focus Delphi: Round II 

In the second round, respondents assigned dates to the range of con 
jectures generated in the previous round. They were asked to write in 
two dates for each event: 

a. The earliest possible date for the event's occurrence; 

b. The date by which time the event most probably would have 



The respondents were asked to restrict their dating conjectures to a 
period covered by the next fifteen years. Additionally, they were given 
two dating options: 

c. "Later" if they felt the event might occur, but at 
some date beyond 1985; 

d. "Never" if they felt the event would never occur. 

Respondents were also given a blank time-line space on which to mark 
the second (most probable) date ("b" above). The coordinators while de- 
signing this round had not anticipated making use of a computer to help 
analyze the data when the forms were returned. Computer analysis was 
soon seen to be significantly more feasible, however, and subsequent data 
was analyzed in this fashion. 

The respondents' "most probable" conjectures were analyzed by a pro- 
gram specifically designed to print out various aggregates of the infor- 
mation and group inter-quartile ranges for each event. The Round III 
response form was generated from the print-outs of the four inter-quartile 
ranges . 



already occurred. 
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Focus Delphi: Round III 



The Round III questionnaire (see Appendix C, p. 59) contained three 
"bits" of information: 

a. The events generated by Round I, aggregated 
and edited by the Delphi coordinators; 

b. The respondent's own original "most probable" 
date estimate from his Round II questionnaire; 

c. The four inter-quartile ranges — one per group — 
for each event, with the median date for the entire 
group specified. 

Respondents were asked for five "bits" of new information: 

a. A new date conjecture if they felt their original 
estimate was inaccurate; 

b. An explanation of why, if their date for any event 
was outside their group's inter-quartile range, 
they felt this date was more accurate; or, why 
they changed their date if they chose to do so; 

c. What they considered the value of the presumed 

occurrence of the event would have for them, 
personally, on five point scale (++, +, 0, — ); 

d. What they considered the value of the presumed occur- 
rence of the event would have for society in general, 
on the same scale as "c" ; 

e. An explication of what the specific consequences would 
be for society in general, again presuming the occur- 
rence of the event. 
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The coordinators felt that a differentiation between personal values 
and value for society was important to focus the respondent’s concern 
on specific impacts of each event's occurrence. Had there been more time 
available between rounds, the value of this could have been compounded 
by asking participants to explicate personal consequences as well as 
societal ones. While the data may not have been directly useful to the 
study, the heuristic exercise it would have afforded the participants 
would have been valuable. 

When these questionnaires were received, the new dates were fed into 
the computer, as were the personal and sc .rLatal value data. The "conse- 
quences for society" statements were collated, refined, and rewritten to 
purge them of over-pejorative implications. 

Focus Delphi : Found IV 

The Round IV response form (Appendix _ s p. 65) reported back to 
respondents five pieces of information: 

a. Twenty of the 1 original sixty-three events; 

b. Adjusted inter-quartile ranges based on Round III dating; 

c. Personal value percentages, disaggregated by groups, on 
a three point scale (+, o, - ) ; 

d. Societal value presented in the same way as (c) above; 

e. The edited "consequences for society" from Round III. 

The twenty events to be included in the fourth round were chosen on 
the basis of four criteria: 

a. The events of the most positive value for society, based 
on the Round III responses to the societal value question; 

b. The events with the most negative value for society; 



c. The events with the highest convergence between groups 
with reference to its date of occurrence; 

d. The events with the least convergence. 

The coordinators felt this mix would provide the widest range of 
events for this prototype study. Also, due to the extremely critical 
time limits available for analyzing data and selecting events, these 
criteria would allow computer selection. For specific planning purposes, 
it may be that a panel of judges would further select events, based upon 
specific planning needs. This particular study was, however, not a data- 
gathering exercise, but a design prototype. Thus, it must be stressed 
that the data, while potentially interesting, were collected only to 
refine a process, not for planning purposes. 

In this last round, respondents were asked to provide two pieces of 
information: 

a. Their perceptions as to which of the four groups had 
the power to enhance or inhibit the occurrence of each 
of the twenty events; 

b. What strategy the respondent would pursue — as a member 
of his own group — to either enhance or inhibit the 
occurrence of each event, depending, of course, on his 
perception of its value to himself and to society in 
general . 

The "power group" information was fed into the computer to be aggre- 
gated and analyzed in various ways (which group was seen as having power, 
group self-perceptions, self-perceptions vs. perceptions of others, etc.). 
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Computer Analysis of Focus Delphi* 



The reason for using the computer should be made clear. Enormous 
amounts of data processing are required to obtain maximum benefit of look- 
ing at the differences between groups. While the predictive nature of 
these differences are of dubious value, the differences should be examined 
for heuristic j rposes . A long-range study is clearly called for by 
which the analyses of data are tied to real world events. Such correlation 
like studies could give the policy-maker valuable guidelines concerring 
how differences between groups affect the lives of man. 

Background of . aa lysis 

All computations for Rounds II and III of the Focus Delphi study 
were done on an IBM 360/50 computer, through a 2741 typewriter terminal, 
using APL. There was a slight storage problem since an APL workspace 
has only 32,000 bits of storage; 4 such workspaces were used to store 
all data and programs. 

Whenever possible, fixed point arithmetic and storage were used. 

Total CPU time for both rounds, including writing and debugging of pro- 
grams, was 30 minutes, which is not unusual for APL, a relatively slow 
interpretative language. The program is available from the Educational 
Policy Research Center at Syracuse. 

There were 65 respondents to the Round II and Round III questionnaires 
answering questions for 63 events.. For Round II, "most probable date" 
was the only statistic worked with, while for Round III there were three 
statistics gathered and compared: changes in the most probable date. 



Special credit must be given to Mr. Michael Folk for his design of the 
computer program for this study. 

22 

25 



"value to you," and "value to society." In Round IV, the only informa- 
tion entered were the various perceptions of power to era_ance or inhibit 
the occurrence of specific events. All were entered at vhe APL terminal. 

Statistical Considerations 



Round II . The Round II statistic used was "most p obable" date o. 
occurrence of each of the 63 events. With responses lit 2 "later" and 
"never" acceptable, it was impossible to consider most cf the usual meas- 
ures of central tendency, so we chose to examine only median and inter- 
quartile range. Any event whose predicted date of occurrence (henceforth 
referred to as PDO) was 1990 or later was automatically considered as 
"later" unless stated as happening "never." For example, if for an ever: 
the median falls in the interval 20 to 47.5 years from now, the event cm 
be considered as having a PDO later than 1990, but not "never"; nothing 
further is implied. 

The median and inter-quartile range for each event was computed for 
each group; then the computer generated a graphic display for each event 
of the four different inter-quartile ranges and medians (see sample Round 
III questionnaire. Appendix C) . No other statistical computations were 
done on the Round II data. It was noted that for some events, groups 
disagreed by as much as 15 or more years on the median PDO. 

Round III . On the third round, participants were asked to observe 
the medians and inter-quartile ranges for each event , generated during 
Round II, to reconsider their original guess, and to revise it if they 
chose. They were asked to judge the value of each event’s occurrence 
to themselves and to society on a -H-, +, 0, -, — scale and to write 
their opinions of the consequences of each event's occurrence to society. 
The first three types of response, which we will call "date forecast," 





’’value to you,' and "value to society," provided quantified data of which 
statistical analysis is possible. 

Some Findings fr^m Round III 



D ate Forecasts . Agreement among groups on median date forecast 
ranged from complete agreement to complete disagreement* The median 
interval between the earliest and latest median forecast for an event 
was 4.5 years Thirteen events had all four medians in a two-year inter- 
val, nine of these were forecasted to happen by 1976. 

There was some indication that there was more agreement on events 
with an earlier PDO, but there were exceptions to this. Only two of 
the forty events whose medians fell in the shortest intervals were fore- 
cast to occur later or never, while all of the twenty events whose medians 
fell in the longest intervals were predicted (by some group) to happen 
later or never. 

The average of the four medians for each event indicated that 55 
events would happen by 1985. The average 75th percentile indicated 39 
events would happen by 1985; the 25th percentile had 60 events happening 
by 1985. 

There was no event which everyone (even everyone within a single 
group) predicted would never happen. In fact, for events whose entire 
inter-quartile range fell in the interval "later-never ," there always 
seemed to be several who thought they would happen within one or two 



years . 



Char. In Date Forecast .* Respondents were asked to revise the 
PDO estlaiE ^ which they had made in Round II. Only 2 major changes in 
median za~ Precast occurred, both by one (or "the same") group, which 
we note ~ly 8 participants and therefore is probably statistically 

insignif t 

In ..1 the inter-quartile range decreased, as should be expected, 

but the i~ . ise was not dramatic. The greatest shifts in median forecast 
occurred events forecast within the decade. Moreover, eight of the 

ten greate.- shifts were toward sooner occurrence of the event. 

If each group is given equal weight, the overall average change in 
median is +.26 years**; the overall average per cent change in inter- 
quartile range is -1.68 years, or a decrease of 9%. Only 4 events had 
an increase m inter-quartile range, while 54 inter-quartile ranges 
decreased. The greatest growth in inter-quartile range was 5 years; the 
greatest decrease was about 14 years. 

Value .resessments . Each person gave estimates, for each event, of 
the value of the occurrence of the event to society and to themselves . 

For computation, the value scale was 1-5, where 1 stood for — and 5 for 
++. A ve assessment of 3, then, indicates neither negative nor posi- 
tive value. Some findings follow. 

Value to Society . Generally, events were seen as of positive value 
to society. Group 4 had the highest percentage (78%) of positive value 

* 

It should be pointed out that between Rounds II and III the nation experi 
enced its first national student strike. The effect of this event on 
any ch? ne or non-change in date forecast or value assessment is not 

within » scope of this study, but might well have been significant. 

** 



We suggest chat the small size of group 2 would render this insignificant 



responses; group 3 had the lowest percentage (70%) (Table 10). Group 4 
had 3 events on which all agreed were of ++ value. No other group had 
total agreement on a ++ value. In fact, group 4 seemed to exhibit most 
agreement . * 

Five events were seen by at least three of the four groups as among 
the ten most valuable to society. Seven events were seen by at least 
three groups as among the ten least valuable. There was little evidence 
of a correlation between positive value and median PDO, but we suggest 
that more negative events are predicted to occur in the later years. 

Value to You . In general, events were seen as being of positive 
value to the individuals. However, no group saw any event as being as 
valuable to themselves as it was to society. Group 3 had the highest 
percentage of positive responses (57%) , while group 4 had the lowest 
percentage (49%) . We note that this is the opposite of the result for 
value to society. Groups 2 and 4 both seemed to exhibit the most agree- 
ment. 

There was little or no evidence of correlation between value to 
self and median PDO. 

General . Group 4 seemed to see the events as of more value to society 
and less value to themselves than any of the other groups. Group 3 seemed 
to see the events as of less value to society and more value to themselves 
than any other group. Group 4 saw fewest events as insignificant to 
society and the most events as insignificant to self. 

Four events were seen as of positive value to both society and self. 
Two events were among the ten most valuable to both society and self for 

* 

See Round IV quest?lonnaire. Appendix D, for results by event and by 
group . 



every group. Six events were considered of negative value to both society 
and self. All groups had the same three events as among the ten most 
detrimental to both society and self. 

There was no strong evidence of a correlation between value and 
median estimated time of occurrence. 

For 12 of the 20 events in Round IV there was better than 67% agree- 
ment as to which of the several groups had the power to cause its occur- 
rence if they pursued the event as a goal. 

In all 20 events (in Round IV) either group 3 (industry) or group 4 
(policy-makers and advisors) were perceived of as having the power to 
cause them to occur. 

Note : A fifth round is essential to create a useful chronology 

of events. The temporal data would, it is assumed, be 
radically reordered with the availability of the value 
information and the perceptions of power. While the 
available data could have been used to construct a 
bogus chronology, the information would have been so 
misleading as to have made such an effort a disservice. 

We wish to re-emphasize that had there been more time and money many 
different statistics could have been computed, many different comparisons 
could have been made. At the least, statistical significance tests 
should have been run on those correlations which were remarked upon. 
Considerably more time and resources would be required to examine thor- 
oughly the different approaches for a Focus Delphi study and to determine 
the best. 
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Designer Comments on 
The Focus Delphi as a Planning Aid 



The Assessment of Goals and Strategies against Future Events 

The Focus Delphi may be useful to help planners examine their goals 
against the perceptions of interested publics concerning those goals and 
other intervening events. 

Specific Areas of Attention 

Events posited to occur in the future may be thought of as goals if 
an aggressor for their occurrence can be specified. If an event is the 
institution's goal, its perceived value to several groups of society 
can be examined by those subsets of the society and duplicated for the 
planners. An array of consequences for the aggregate society can also 
be examined. 

Value to Planners 



Information gathered by a Focus Delphi can be examined a number of 

ways : 



- The perceptions of the date of the occurrence of any event can be 
examined against perceptions of power to cause the event. If, for 
example, all groups estimate an event will occur by 1975 and the same 
group is seen by all as having the power to affect the event's occur- 
rence, but they do not care to aggress for the event, the event probably 
will not occur because the power group will not pursue it. 
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- The repeated strategies of each group can be examined to see the 
array of behaviors that the planner might be forced to confront while 
pursuing his strategy. 

- A planner can examine his strategy through time against dated 
intervening events to examine where and how impacts might occur. 

- Emerging future events beyond institutional control that may 
impact on the planner's goal can be identified. 

- Events can be examined against a strategy, and a contingency plan 
to forestall negative impacts can be prepared. 

- The consequences of an event, anticipated by the participating 
groups, can supply the planner with previously unforeseen outcomes to 
substantiate or modify his case. 

- The value of any goal to individuals can be weighed against the 
value to society in evaluating the goal. 

- An examination of strategy responses can improve understanding 
of the political attitudes of various groups of the society. 

- To examine the amount of planned goal attainment, consequences 
are weighed against event dating to assess probable consequences if the 
date were shifted. 

- In summary, then, the questions a planner might ask while examin- 
ing his goals are: 

- What events have impact on the primary goal? 

- Are these events goals in themselves? Of whom? 

- When do various groups anticipate it will occur? 



- Of what value is the event to each group? Individual? 

- Of what value is it seen as being to society? 

- Of what societal consequence? 

- Who is thought to control the event? 

- What can be done to help or hurt it? 

- Can the events be redated to modify impacts 
and perception? How?* 

- Where dissensus exists between participating groups, the planner 
can get a sense of the various opinions, related directly to the nature 
and/or description of the event. Can the same event be stated another 
way to change attitudes about it? 

By systematically focusing on the questions raised here, a planner 
can begin to cope with the array of concerns and reasons for concern 
demonstrated in various sectors of society. In the prototype study, 
for example, it is interesting that both the high school (input) and 
college (throughput) groups felt they do not have very much control. 
Some control may need to be given them to increase their concern and 
interest in any given future event. 

The purpose of the Focus Delphi, then, is to help planners assess 
their strategy options against the beliefs, attitudes, and consequences 
felt and seen by various members of their society to help determine the 
value of planned goals. 



This is a modification of our first experiment reported here, 
in use in all our subsequent contract work with the tool. 
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It is 



INTERLUDE 



Posit an institutional planning group whose members now understand 
the implications of the pursuit of their long-term goals. Now they must 
examine the entire array of goals to determine priorities. Goal state- 
ments are, at best, only understood as cliches. To rally support and 
justify pursuit of his goal, the planner must make clear to the institu- 
tion the implications of the goal statement and how the goal impacts and 
is impacted upon by the pursuit of all other goals. 

We must examine the impact of these goals. Certain goals will be at 
cross purposes with each other. We examine these cross purposes with 
the aid of the C r o s s -P ur p o e A : Matrix. 



PART II 



THE CROSS-IMPACT/CROSS-PURPOSE MATRIX; 

A TOOL TO HELP SET INSTITUTIONAL PRIORITIES 
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THE CROSS -IMPACT /CROSS-PURPOSE MATRIX; 

A TOOL TO HELP SET INSTITUTIONAL PRIORITIES 



The Cross-Impact Matrix 

The Cross-Impact Matrix is a computer program designed to rectify 
a major flaw in Delphi (discussed earlier). The flaw had been recognized 
by the designers of Delphi and the fact is that events in a Delphi are 
dealt with as independent entities. No notice is given in Delphi to the 
possibility that the occurrence of one event will alter the probability 
of the occurrence of a subsequent event. Of all the issues facing fore- 
casters the problem of event interdependency is probably the most vexing 
(Pardee in Gordon-Hayward, December 1968, p. 101) . It is hard to think 
of an event for which no predecessor exists in some causal manner (Pardee, 

p. 101). 

A general theory of cross impacts does not exist (Gordon-Hayward, 
December 1968, p. 101). The Cross-Impact Matrix program attempts to deal 
with this problem. The process is rather simple. Events drawn from a 
Delphi are examined against an arbitrary time horizon to assess their prob 
ability, plausibility, and impact. An additional problem is caused by 
the different nature of various "future events," as discussed in the pre- 
vious section. Both the Cross-Impact Matrix and Cross-Purpose Matrix 
deal with future events. The Cross-Impact Matrix deals with that subset 
of future events called "future news events." The Cross-Purpose Matrix 
deals with that subset of future events called "goals." 





However, a continuing problem not yet effectively dealt with is the 
substantive nature of the judgments required when filling in the cells of 
the matrix. There is ample evidence that the quality of output from a 
Cross-Impact Matrix is very much determined by the quality of analysis 
and judgment which goes into filling the cells (Ziegler, May 1970, p. 99). 
Further, Michael Folk, working on the mathematic formula used in the 
device, finds serious and apparently unresolvable problems with claims 
made for the device (Folk, 1971). These dissatisfactions led to the de- 
sign of what is called the Cross-Purpose Matrix. 



The Cross-Purpose Matrix 



The Cross-Purpose Matrix was devised to deal with goals pursued by 
advocates within an institution. A goal is an event that is intended 
by someone to occur. The Cross-Purpose Matrix attempts not to replace 
the Cross-Impact Matrix, but to deal with a subset of future events, 
i.e., goals, that cannot be assessed in the same way as future news 
events. The first application of the Cross-Purpose Matrix was in the 
MIDSIM exercise.* 



The Cross-Purpose Matrix has several operational rules: 1) There 

is an identified aggressor, i.e., advocate, for each goal. 2) Original 
probability information concerning the impact of other goals on his — 
and vice versa — as reported in each of the matrix boxes can be determined 
only by the person aggressing for the goal under consideration. 3) The 
full meaning of a goal, its impact on each of the others, and the impact 



* 

The Maxwell International Development Seminar was a 30-day re-orientation 
program for staff officials of the Agency for International Development 
of the U. S. State Department on overseas assignments in several areas 
of concern. The Educational Policy Research Center at Syracuse con- 
tracted to construct a planning exercise (MIDSIM) to help focus partici- 
pants’ attention on long-term planning. 



of each of the other goals on it depends upon how the aggressor intends 
to behave in pursuit of his goal. This strategy, since it has not yet 
been specified, is known only to him. Thus, the subject of discussion 
between advocates is their planned strategies. The examination of the 
goals themselves is inadequate. The very process of grouping under a 
single goal all possible ways one might behave in order to reach that 
goal tends to hide the way one will behave. Unintended consequences and 
strategies which negatively affect other goals are thus often unexplored. 
What must be explicated and assessed are the activities and events that 
comprise a sufficient strategy to bring about the goal’s attainment. 

These basic conditional rules underlie the Cross-Purpose Matrix. 

To prepare his goal for consideration the planner must answer several 
questions : 



1. "If the institution adopted my goal as its top priority, 
when is the earliest possible date it could occur, knowing 
the time necessary for each of the activities and events 
that comprise my proposed strategy which is sufficient to 
force the goal's occurrence?" 

2. "I know competing goals exist within the organization, all 
vying for limited energies. I know them only as cliches, 
but some are probably as worthy as mine. Knowing this, 
and nothing more about the other goals, when is it most 
plausible to assume my goal will become an occurred event?" 

A matrix is constructed with all goal statements under examination 
listed both vertically and horizontally at the side and top of the matrix 
(see Figure I, Appendix E, page 71). Events are listed chronologically 
from top to bottom and left to right, based on the most plausible esti- 
mate of the occurrence date (question 2) set by the goal advocate. 





Advocates for each goal complete a portion of each row and column 
ar»propr _r i.te to the_r goal. The process is as follows: When answering 

horizontally across the matrix (row) , each responds by asking himself 
for his goal with respect to each other goal this question: "I assume 

I am going to be 100% successful in attaining my goal. If so, what will 
the effect of my success be on the hoped-for success of each of the other 
goals as I understand them?” Responses are marked in the bottom half of 
each square in the appropriate row in the matrix. The designation of 
impact is ++, +, 0, — . 

The same goals are approached by each individual vertically (column) 
on the matrix, and responded to in the top half of each box; to respond 
he asks himself this question: "I assume their goal is 100% successful. 
What will its success do to me and my hoped-for success?" He again re- 
sponds with a ++, +, 0, — scale (see Figure I, Appendix E, in which 

hypothetical goal # 4 has the appropriate halves of each cell marked) . 

Individual responses are then displayed on one common matrix, all 
goal advocates' responses being visible to all others. The focusing mate- 
rial necessary for goal assessment through strategy specification is now 
displayed. The positive and negative impacts of the various goals, as 
seen by all the individual goal advocates, is visible. This display 
demonstrates the various perceptions of each goal and how they are affected 
by others' goals. This exercise compels planners not to consider their 
goal as given. 

In any box in the matrix, a spread of opinion is displayed (see 
Figure II, Appendix E, page 72)* The bottom half of any box along the 
horizontal (row) shows the goal advocate's view of the impact of his pro- 
gram's success on the hoped-for success of the goal listed in the verti- 
cal. The top half of the box shows the same information, but from the 
perceptions of the goal advocate of the vertically (column) listed goal. 
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That mi assume a horizontally listed goal, "a," and a vertically listed 
goal, . " Their advocates are indicated by A and B respectively. In 
the ::::ttom half of the box, A indicates his view of the effect upon "b" 
of the total success of "a," while in the top half is B’s perception of 
the goal "a" upon "b" — both ask the same question, but the answers come 
from two different perspectives. Each box, of course, has its alternate, 
where the reverse question (the effect of M b M upon the total success of 
"a") is asked by each participant. Though the information spread can be 
analyzed, this exercise is best used only to move the planning group for- 
ward to a more important stage of the exercise. 

Individuals or groups now attempt to discover the reasons for the 
spread of opinion in any given box. They do this by discussing the full 
meaning of the goal, its implications, and the strategy the advocate sees 
as sufficient to attain the goal, i.e., "what does this cliche mean?" 

"how do you propose to pursue your goal?" The process continues until 
each cubicle of the matrix displays only one bit of information. That 
bit of information is the agreed~to perceptions of both goal agressors as 
to the probable impact (-,0, or +) of one goal on the other. 

We might now add to this matrix of goals various "news events" that 
comprise one possible future environment over time. Thus, the participant 
is asked to respond not only about his goal with respect to all other goals , 
but with respect to news events beyond direct institutional control: 

- "If my program is successful, what is its impact on the 
occurrence of the news event?" 

- "If the news event occurs, what does it do to the 
anticipated success of my goal?" 
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Analysis 1 



At this stage it is appropriate to ask questions about the information 
displayed. The responses represented in the matrix by + and - symbols 
give an array of perceptions about the impact of the pursuit of each of 
a set of goals on all other goals of that institution. 

First, in any cubicle of the pre-negotiation matrix (where a spread 
of opinion may still exist) some cubicles may have a common impact sign 
(+/+). However, that does not mean there is common perception of the im- 
pact. The same discussion concerning goal implications and strategy must 
take place: 

- Perceptions of participants may have been based on a misunderstand- 
ing of the goal; 

- The goal may have been too abstractly described to accurately 
reflect expected changed behaviors foreseen by its advocate; 

- The idea may have been too novel to be understood without 
its advocate’s rationale and the value shifts it implies; 

- The goal may be agreed to, but the sufficient strategy may 
alter perceived impacts. 

Further, the dynamics of the negotiation process itself can be 
examined: 

- What is the nature of the negotiations? That is, "Who 
' moves?” "Who seeks whom?" 

- What is the direction of agreement when they finally 
arrive at an impact? Is it toward the affected or the 
af f ector? 



- Why did agreement occur (if it did)? 

- Was change based on rational or other grounds? 

Any institution contains sectors of interest and of different subject 
concerns. One can examine all the goals in a single sector of interest 
against the others. This helps specify, within any individual sector of 
interest, a top priority program within that sector. One might also 
examine the programs of one sector against the programs of one other 
sector, examining whether the sectors interact or if any programs within 
them affect each other, i.e., the goals of the research departments vs. 
those of the engineering department. 

One can examine the highly negative cells and the highly positive 
cells — that is, those with a — or ++, both before and after negotiation. 
What is the nature of the high impact perceived by both parties? One 
might also analyze the spread, which may be quantified on an absolute 
scale. The spread between -H- and — in any box is a relative misunder- 
standing of 4; that is, there is a spread of five digits between those two. 
The difference between + and 0 is an absolute spread of 1. Absolute 
spreads can be quantified to examine the magnitude of misunderstanding 
between certain sectors. That is, does the terminology and value of a 
sector tend to be regularly misunderstood by any other sector? 

Next, one might examine the general range of impact of each goal: 

- What are the goals that consistently have no perceived impact 
on any others? 

- Which goals and their strategies tend to be stable, that is, unaf- 
fected by those external news events mentioned above? 

- Which tend to be unaffected over-all by the occurrence of many 
news events or the occurrence of all other programs? 



Those that are apparently unaffected by events we might call stable 
goals. An unstable program would be perceived by the participants as very 
sensitive to the occurrence of other events. 

We might examine at what moment a goal is unstable and at what moment 
in time it becomes stable. For example, a new government agency languishes 
on an unstable state where nearly any news event affects it until it re- 
ceives its funding. From that moment until the next funding request, it 
is largely a stable agency. It would be perceived as such and would be 
visible in the matrix as such. This kind of analysis allows the execu- 
tive to calendar those things he feels are fragile at various moments, 
rather than to worry about everything all the time. Another examination 
one might make is to focus concern on those programs within the matrix 
which show consistent value for all other programs, positive or negative, 
at the moment when they are planned to occur. 

These comments, then, explore some possible uses of the Cross-Purpose 
Matrix as an interaction device for long-range planning. The Cross- 
Purpose Matrix represents a confrontation focusing device, which compels 
planners to array their strategies before competing advocates and to 
defend them. The strategies reflect planned behaviors over time. The 
interactions of these behaviors with the outside world build a material 
base for the set of events that occur and affect all lives in a society. 

The exercises described above present a logical set of questions that 
long-term planners might reasonably confront in preparing to alter the 
future. 

The future environments we will actually encounter will include the 
results of the actions of present men, pursuing what they believe to be 
worthwhile goals. Men can thus carefully examine future plans in a future 
environment (one that is comprised of a set of events they agree are plaus- 
ible) . Men believe in what they do. In the planning stage they must 
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seize the opportunity to suspend their disbelief and to examine their 
ideas in dispassionate terms of sufficiency. 



* 4c * 

AUTHOR'S NOTE 

Throughout this report the underlying assumption has been that we 
must see the future as filled with an array of human goals. To study 
and examine a future, an individual most consider his examination as 
merely sufficient. When priorities are specified and goals pursued, any 
sufficient future becomes linked to the present. Men must examine their 
sufficient plans before these plans are imposed on an unprepared world. 

When futurists and policy-makers use techniques that induce consen- 
sus, or conformity to an opinion, each individual's ability to believe 
is threatened. Those who disagree with the consensus find the foundation 
of all their thoughts about the future weak and undermined. Consensus 
attempts to induce closure. Too early closure prevents suspended dis- 
belief and compels us to operate as if what we do were necessary rather 
than merely sufficient, inevitable rather than a matter of choice. 

Only sustained divergence allows a mote thorough examination of plans 
to exist, yet divergence becomes more difficult to sustain as our play at 
futures grows formalized. Convergence and consensus methods reach closure 
but eliminate alternative future possibilities. 
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APPENDIX A 
FOCUS DELPHI 
ROUND I MATERIALS 




FOCUS DELPHI— ROUND I QUESTIONNAIRE 



In the spaces provided below, list ten even ts that you conjecture 
might take place during the next fifteen years that you perceive may 
have some impact upon electro-mechanical technology and education. In 
your conjectures, think about things that might happen that would affect: 



- administration of schools 

- industrial training programs 

- manpower needs of the industry (in terms of number and capacity) 

- instructional methods, i.e., how could what is taught be taught 

more effectively 

- curriculum, i.e., what is not taught now that will have to be 

sometime during the next fifteen years 

- industry hiring practices 

- practices concerning admission to educational or training pro- 

grams 

- technical advances that will seriously affect both industry 

and their educational inputs 

- nature of student population (racial, sexual r- • ios, etc.) 

Events might be: 



- new legislation 

- policy decisions 

- occurrences 

- technical breakthroughs 

- major shifts of opinion and attitudes in various populations 

(students, personnel in charge of hiring, teachers, ad- 
ministrators, the general public) 

- changes in curriculum or instructional methods in the schools 



Do not feel constrained to placing dates on the events, or placing 
the in any chronological order. Simply list ten events that you con- 
jecture as being plausible during the next fifteen years. 
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FOCUS DELPHI— ROUND I RESPONSE FORM 



Event 1 



E" van t 2 



Event 3 



Event 4 



Event 5 



Event 6 



Event 7 



Event 8 



Event 9 



Event 10 



O 

ERIC 



Name. 



(Please print) 
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APPENDIX B 



FOCUS DELPHI 
ROUND II MATERIALS 




SO 



Dear 



Thank you. for your prompt response on the first round of the Focus 
Delphi. The events in this second round are drawn from your lists of 
conjee tured events. 

All of the events on this questionnaire were generated by panelists. 
We have taken events that occurred repeatedly on the questionnaires, that 
seemed particularly interesting, or could have high impact for education- 
al planning. These events were then grouped into three categories — Edu- 
cational, Industrial, and Technological Breakthroughs — and rewritten. 

The rewording of some of the events was necessary to allow them to be 
read with as little misinterpretation as possible, and to remove from 
them as much "value weighting" as possible. 

The purpose of this round is to assign dates to the events. You. 
are asked to conjecture about two dates: Most Probable date of occur- 

rence (MP) and Earl5.es t date of occurrence (E) . 

In other words, speculate as to when you think the event listed will 
Most Probably have already occurred. It might be helpful to think of 
this in terms of percentages , so think of it as 'By what date will there 
be a 90% chance that the event will have already occurred?" 

Likewise, conjecture about the Earliest probable date by which you 
think each event can occur. 

Place these dates in the appropriate columns next to each event, 
being careful to place the Most Probable date in the column "MP" and 
the Earliest date in the column "E". 

Then, in the spaces marked for the time continuum, place a dot 
approximately under the date you have chosen as the Most Probable. You 
will notice that as the time indicated is further Into the future, the 
amount of space al_..cted for each year decreases. This is because it 
has been found through a number of studies that the further people con- 
jecture ixrto the future, the less precise their conjectures become. 

Thus, it is not necessary to be as precise in the placement of the mark 
for the years 1981-1985, as it might be for 1973-1976. 



You will also notice that columns have been marked "Later* 1 and 
"Never." If you think that the event will occur AFTER 1985, but that 

it will occur, place a mark in the "Later** column, and place an "L" in 

the "MP." This, of course, does not mean that it might not occur be- 
fore 1985, so you can place a date in the "E" column. 

If you think the event will never occur, for any reason, place a 
mark in the "Never" column, and place an "N" in the "MP" column. 

At the end of the list of events, one page has been left blank in 

case you would like to add additional events to the list. Write your 
events in the appropriate place, fill out the other columns as instruc- 
ted above, and return the questionnaire. If a number of people add the 
same or similar events, these events will be included in subsequent 
rounds . 

Please return the questionnaire in the enclosed envelop within two 
days after receiving it. We very much appreciate your prompt response 
and will return the next round as soon as possible. 

Thank you. 



DeLayne Hudspeth 



DH: cm 
Enc. 
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FILMED FROM BEST AVAILABLE COPY 



HI - ROUND II RESPONSE KORM 



» -•* 1 i ON 



EVENT 



MP 



L 



]. Govern jsidi.es to institutional programs of 

el;- cc 4 .nical education increase 50% over 1968 
levels due to skilled manpower shortage. 



. Federal legislation requires administration to 
ut.'&i much more authori tiveiy with disrupcors of class- 
r o oir. s . yii ni:aun p unis lime n t : one s ome seer sus pens ion . 



(19 70- l 1973- 

fl972 j 1976 



1977- 

1985 

A t JUJ-1 j J. 



I 

l-'nite students rejected from programs with rapidly \ 
growing black student population (yearly increase 50% 
or more) take legal action against schools and programs I 
to show cause. j 

i 



4. Veiv.nl e enrollment in previously predominant male 
t e c h n L c ax prog c ains double 1968 level- 



ir ogyams that allow students to leave and re-enter 
f.ducat j oru.l programs without penalty instituted at many 
schools and recognized by industry. Many students work 
-ft the sympathetic industries between school stints 
tfltri no work or pay penalties. 



1*. Federal government allots six million dollars a 
year to pLan, develop, and encourage innovational 
occupational programs of education. 



7. Technical and vocational courses removed from 
academic centers and concentrated into area centers 
vita close contacts with local and large industry. 



S. Soaring college costs cause three fold increase in 
vocational program enrollments . 



?. Engineering colleges will demand that high schools 
taxe over full instruction in calculus, computer pro- 
and industrial gr;iphics. 



4 . C. .4(1 . 1 . L 
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Dear Respondent: 

Enclosed you will find Round III of the Focus Delphi Study. Al- 
though at first glance it may seem formidable, it should take less time 
than the first two passes to complete. 

Explanation of the Data Display 

In this pass, you will find displayed the data you ventured to us 
on the last questionnaire. The inter-quartile range of the data (the 
middle 50% of the range of dates) is displayed with dash lines. For 
example, for event # 3, group #1, the interquartile, range of dates is 
from 1971 to 1981. 



1970 75 80 85 90 Later Never 



For group #2 in event 3, the range is from 1976 through N (never). 

The median date (the date above which one-half of the respondents 
conjectured) Is indicated with an arrowhead. Again, using event #3, 
group #1, we find that 50% of the respondents gave us dates above 1974, 
and 50% responded below 1974. 



1970 75 80 85 90 Later Never 



As you can see, the range of responses varies widely — sometimes by 
"group," and sometimes widely within a single group. 

Groups 

The total population involved in this study can be broken into 
four groups : 

Group 1: Teachers, counselors, and administrators of New 

York State high schools. 

Group 2: Teachers and administrators of two-year vocational 

schools with programs in electro-mechanical tech- 
nology. 




b 



59 



Group 3 : 



Employers of graduates of electro-mechanical pro- 
grams and recent graduates of such programs cur- 
rently employed. 

Group 4: Personnel at various levels of the New York State 

Department of Education and Vocational Training 
and others affecting educational policy. 

Written on the upper left hand corner of the questionnaire you will 
find your name with a number (1-4) . This number indicates the group in 
which you are placed, even though some participants could fit into more 
than one of the categories described above. 

Task 

We are adking you to do three things on this pass* (1) examine the 
relationship of the date you provided us with the range of dates from 
the four groups; (2) assign a value estimate to the event ; and (3) 
write a very short description about the possible consequence of the 
event in question. 

1. Your original date estimate is in the "MP" (most probable) 
column. Compare your date to the Interquartile range of your 
group and the other groups as defined above. If, upon recon- 
sideration, you would like to change your estimate, put the 
new date in the "New Date” column located to the right of the 
"MP" column, and explain briefly the reasons for the. change 
In the "Consequences for Society" column, ("L" would be used 
for later than 1990 and "N" for never) . If you feel your 
original estimate was best, then put a check mark in the "New 
Date" column , 

2, After looking at the date of each event, go to the column' 
marked "Value to You." In this column please respond to the 
following question* "Assuming that the event in question 
WILL OCCUR, what will be its effect on me personally?" Use 
one of the following symbols; 

++ High positive value 
+ Positive value 
0 Little effect, if any 

- Negative value (it will hurt me, but not much) 

— Extreme negative value 

For instance, if an event were to occur that would substantial- 
ly raise your standard of living, however you personally define 
that, it would be a ++ event; if however s the occurrence of 
the event would mean that you would lose your job and would 
make you unemployable, it would be a — event. 
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3 , 



Next, and irrespective of how the event: would affect you per- 
sonally , think about the consequence of this event upon society 
as a whole. Assign the same value sign of ++ (high value) to 
— (negative value) , put this symbol into the ''Consequences 
for Society" box and then write a terse statement describing 
the nature of this consequence. (E.g,, it would improve the 
quality of education; or, it would contribute to pollution). 

Many thanks for your patience. The next and final round will have 
fewer items and be much less complex. Please take the time to complete 
this within the next two days . Your honorarium will be mailed within 
iwo days after the final questionnaire is re tuned. 



Sincerely, 



DeLayne Hudspeth 



DH: cm 
Enc , 



•ft 



-) 



-j 




i 

5 

U 

O 

• 

V 

u 

ft 

• 

* 

I 














• 




1 




ii 

a © 
























o 

rl 


rl 




<4 

N 


rO 

H 


=T 

H 


k/1 

N 




to 

*4 


is 

rt 




CO 

M 


a* 

rt 




i l 
1 ! 

* * * 






1 

i 

i 


• 


1 

i 

i « 


1 

1 | 


i 

( 






1 

1- J-. 






| 1 “I t 

IIII 
(III 

l fe a i. _ 




1 

j 


i*i i 
i ( f 
i i - 


1 

« 




<* 

1 

* 


i • 

- | 


^ -<3 

i t 

j § 


i i 

i i 

| i J 


1 

- > 






<1 








— i *— 4 4 

^ ^ ^ ^ 
(III 


\ 


i 


; i 

i i 

i i 

1 * * 


i 

i 

i 

| 




i 

e 

i 

1 

i — 


1 

1 

i 

i 


' i i 

i i 

1 

— ^ 




i i 

II 


i 














| | J j 

IIII 

IIII 

IIII 

- - ■ * - * 


\ 


i i i 

: i i 

f i i 

-_i ii 


I 

i 

i 




i 

i 

i 

I 


i i 

s s 
1 1 
ill 


i 

i 

! ! 




1 1 

1 1 

1 1 

_ _«_ 1 _ 
















IIII 
( II, 

1 1 1 
1 f 


\ 


<? i 

lilt 
i t i 

i t i 


i W 

S S 1 

( 1 i 

» i 


i 

! S 

w 


i t 

t i 

t I 

I I 

i- - - I 


*a i i i 

1 1 l<3 

i <j ii 
till 

i l-t 1 


t i I 
I I I 
M 1 

1 1 » 

» 




<1 1 
1 1 
1 1 o 

_LL, 




1 

t f 


<J 




s 

u 




1 

1 

i i i 

i i i 


J — 

I 

1 

1 

1 

" 


V 

M 


r ! ! 


' ' ! 


| 1 


<3 1 1 i 

1 r. 


iiii 


i 1 

\ 


t I 

1 


^ i i 1 
i i i<i 
N ^ | 

1 1 
i 


j 




i i 
t 
* 


■a *3 I 1 
1 l*s l 
1 1 i<i 

l II 

. . *• i 




f* $* m a- 


H fi © ^ 


h ^ n ^ 


rt R ^ 


H fl PI # 


N ft « # 


N ft ft 3 s 


sn « m ^ j 


«4 R # 


h ft 


41 

22 

,35 A 












J - ' iL, ^ 








i 

i 


S! 




















i 


N 

I 


. 

a o 

32 

a 

2 & 

5 £ 

m * 

il 

4 8 

*4 
*N W 

11 
i § 

is 

N 

©i 

H * 

« t> 

IS" 

i! 

Is 

l|i 

• b « 

aajL_ 


I 

y 

!&■' 

I 1 

u 

?■§ 

5 | 

a 

g* 

Tt W 

M y 
a h 

U *H 
3 X) 

T3 « 

V H 
*4 i 

u « • 

*-• a a 
■t-o » 

*w a 

a * 9 
U M 
Mu 
i * 8 

^4 y O 

S»g 

END 

7 * a *a 

fe> ii e 

M « 

• & 
Ned 
laJ " ^ 


| 

*r» 

I 

4 

I 

M 

$ 

M 

CW 

N 

€ 

1 

«• 

0 

g 

■3 

y 

***» # 

I 8 
83 

»s 

•p o 
•-* ■ 

* « 


i. 

y m 

u 

v *a a 

MCH 
S I 4J 

a ^ y 
1 

11 * 

M H5& 

& 2 2 
a & a. 

« 

! ° i 
1 S w 

M 3 *4 

"3 S 1 

0 -n e 
a a m 
M S a 

U « -N 

1 u 

ax 

*:i / 

|*5 

y « a 
* &.» 

•S5 8 

M N 

«1 8 
els 


1 . 
Ii 

a &o 

s i 

m -h 
in a 
o ^ 
o m 
m m 
a m 
a 

a *3 

* B 

m m 

4J y 

i« 

OC «l 

y 

O «4 
« 

3 5 

a m 

S -8 
5 2 

&I 

y a 

li 

M M 
M 

s § 


% 


■H M 

•1* 
y T3 

3 1 
u a 

Is 

^ 9 

25 

a y 

i § 
E8 
S " 

as* 
at « « 
m *a 
a « « 

yt e; 

a m e 
U rn 
m M w 
« a 
*a ,a s 
g n i 

m** m 

•a sS 

N «M 

a N 0 

^ 15 


i 

■3 ■ 
55 

J • 

88 
« »-> 

9 H 
"H Q 

•9 ^ 

S 2 

4 S 

o -o 

23 

5 ■ 

1 1 
f-i B. 

32 

2 & 

M U 

?Ss 

Baa 

srs 

^ ■ j 

a a 

l! s 

• a a 
M m a 


2 4 

ISI2 
Sals 
M M | If 
M m H 

u 2 4 « 

•1 *t4 

Wi M M 

vox; 

3 8S5 
5S°f 

a e b 
u Dny 
a a a 

m m 6 
x: m 
a « 
e w b ^ 
a & y 

DH N 

5 8 8 8 
1 “fl . 

x: 

> M y W 

i s » a 

« *n *j <y 
m x: 

h B y u 
1 D i 

•■f 6 a a 

oS‘ 

ii i r 3 

u a <n a 

.3 

M i* a 
Q DM Q- 
K m a > 
a a u 
* DO 
h£ub 
-* *« t m » * 


<-l 

a 

2 

a 

0 * 

*a a 
€ a 
■h a 
y 

2 5 

m e 

j= g 

5 8 

II 

y 

3 2 

1 8 

1 • 
g a 
a a 

* 2 
■a a 
e a 

a 

a 

ft u 

a 

u a 

«M U 

J i 

If 

S § ! 


• 

m i 

la 1 

ie4 

5 45 a 
e a u 
*m u a 
a a a 

M > u 
y m 
a e 
a s a 

a -n | 

a » w » 

Mm* 4 | 

D 3 3 • ; 

60 M a 

ty D y j 

a -n m g 4 

^ ^ a 

m a m 
a y u a 

M U u 

y x o s 
is y h 
NM M E 
b a a 

iM a ^ 

D h u ■ 

m N a 

a « m m * 

1 1 u 

y<B £ y 
qua 
DM K M 
*j 

D D N y 

m ^ a y 
^ ^ i 

1 a a B * , 

6 « N g ‘ 1 

□ MW ■ • 

« 1 & 

£% % i 

y n s 



62 




58 



APPENDIX D 



FOCUS DELPHI 
ROUND XV MATERIALS 



ERLC 

imiMiffaHaaaa 



i 




Dear Respondent: 



Attached to this letter are twenty events which constitute the 
fourth and final round of the Focus Delphi, The data displayed on 
these events were provided by you on the last round. We are asking 
you to provide two additional judgments about each event. 

First, let me review the format of the questionnaire. The numbers 
on the time line for each event refer to four subgroups i 

1. Teachers and administrators of high schools involved 
in technical education, 

2, Teachers and administrators of technical programs in 
two-year programs of electro -mechanical education be- 
yond the secondary level., 

3, Employers of recent graduates of programs of electro- 
mechanical education and recent graduates of such pro- 
grams , 

4. Policy advisory persons to the New York State Department 
of Education concerned with technical education. 

Time estimates from each group 'are represented with — • — lines. 

The length of the line indicates the middle 501 of responses (the top 
25% and bottom 25% were dropped). The V symbol indicates the median! 
that point above wlch 50% of the participants placed a date. 

In the next two columns to the right, "Personal Value" and "Value 
to Society," the percentage numbers represent responses to the two ques- 
tions: "What value do you see the occurrence of the event having for 

you personally?" and "What value do you see the occurrence of the event 
having for society in general? " As you can see, these responses have 
been reported back by groups, as defined above. 

The next colunai reports back, collated and edited, specific con- 
sequences for society assuming the event occurs , These have been marked 
as to whether the respondents thought that the consequence was positive 
(+) or negative (-) , In all cases, the positive consequences are listed 
first, the negative second. By reading this information in conjunction 
with the percentage figures in the two columns given to "values," you 
can assess the information. 
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For example, for Event X, Group 1 and Group 2 list their value 
responses as follows: 



1 

2 



Personal Value 

0 + 

10 40 50 

35 65 



Value to Society 

0 + 

20 80 

10 80 10 



This means that 101 of Group 1 saw negative personal value assuming the 
occurrence of the event, 40% saw no impact at all, and 50% saw the occur- 
rence of the event as positive for themselves. At the same time, none 
in the group saw any negative impact on society* 20% saw it as having 
no impact on society, and 80% saw it as a socially valuable event. In 
Group 2, the responses were different. The "Consequences for Society" 
column gives specific reasons for the responses in the "Value to Society" 
column . 



The next two columns, those with an asterisk (*) above them, are 
blank and to be filled In by you. 

Column labelled "P of P": The letters stand for your 

"Perception of Power". Assuming that the event can be made 
to or prevented from occurring, you are to conjecture ''Who 
among the four groups as defined above do you think would 
have the power to cause its occurrence?" When you have de- 
cided which group this would be, place the number of the group 
in the space provided. 

Column labelled "Strategy (Enhance or Retard)": In the space 

provided, and as briefly and succinctly as possible, write 
what you would do to enhance or retard the occurrence of the 
event. When writing, consider three things: 

a. The event's value to you| 

b„ The event's value to society (as you see it) | and 

c. Your perception of who has the power ("P of P") to 

enhance or retard the occurrence of the event (and how 
they see its value to society) , 

If, for example, an event would he of negative value for you 
personally, and you see it as being negative for society (but 
the group you consider to have the power to cause it sees it 
as positive) what strategy might you as a member of your group 
use to retard the occurrence of the event? 

When you have completed these two tasks, return the questionnaire 
in the enclosed envelop. Again, please return it within 48 hours. As 
soon as we receive your questionnaire we will process and mail your check. 

Thank you very much for participating. We think we have an excit- 
ing study and will send you a copy of the final report as soon as the 
last data is analyzed and the report printed. 



Sincerely, 
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PeLayne Hudspeth 
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APPENDIX E 



CROSS -PURPOSE MATRIX 



FIGURE I 
FIGURE II 
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A 1 651 8 -U 




CROSS * PURPOSE MATRIX KEY 

+ + = STRONG POSITIVE IMPACT 

+ = POSITIVE IMPACT 

0 = NEUTRAL OR NO IMPACT 

- = NEGATIVE IMPACT 

STRONG NEGATIVE IMPACT 
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